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Starting DEOS Analysis 


8 One day briefing by HTC at NASA Ames 

- System Desorption 

- High-level description of known Error 

• NASA teem did not knew wharlhe precise error was and 
haw io make it eppear 

• HTC delivered ±3000 lines of C++ cede 

• Model Check Actual Source Code 

- Mode! Extraction Requires Expert Users 

- Goal is to Improve Certification Process 


analysis Results 


Translation required 3 man-months 

- Ch- translation was straight-forward 

- Envirormentde^elcpT>enr took mast time 

Found Error by Checking Temporal Property 

- O(startfteriod "(ierdPenod U idleRun)) 

• The 'die' thread runs when mthir^ eise ccn hence time partitioning 
is violated if idie does not run between the start and end of a specific 
period 

DEOS Team Reaction 

- Surprised that error was found by directly checking code 

- They expected NASA team to ask for smaller “slice' 


Model Checking 


Sms Bsssztsh tenter 

Use SPIN model checker 

Translate DECS C++ code to PROME LA 

- Systematic Translation Process (by hand) 

• l-to-1 Mcppirg: C++ to PROMELA 

Developed Nondeterministfo Environment 

- Model Timer and System Ticks to Remove Real-time 

• Time Modeled by Norxfeterministic Chofce of Values 

- Highly Flexible Model of Threads 

• Thread creation, deieifon and API calls ccn occur dynamically 


DEOS Lessons 


SsmSessstti tester - 


• Model checking at source code level is feasible 

6 Environment creation is hard - - - 

- To this day it is THE problem in model checking 

• Research follow-up study 

- Translated C++ to Java and used 
JavaPathFinder (JPF) model checker directly 

- Showed filter-based environment generation has 
potential 
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Model Checking 


/ Amss Beseem Sealer - 


“Atomic” statements added 

- Although JPF support partial-order reduction, we don’t have a good static 
analysis tool to calculate independent transitions 

- We do now have a version of JPF that groups all transitions between 
synchronization statements into an atomic block 

* Do automatically what we did manually 

• Scott Stoller 

Implemented a “Factory” based infrastructare for adding abstractions 

- Abstractions play such a key role in model checking that we didn’ t want them 
to struggle with engineering issues instead of creating new abstractions 

We gave them the “point” abstraction of time 

- All time-based decisions became nondeterministic 

- It is typical to start with the most over-approximated system and use 
refinement as necessary 

We gave them the “Universal” planner that can create all plans up to a 

specific size nondetermimstically 


static Analysis Setup 


J?/ Arnes Esszmh Sealer- 


The experimental conditions for static analysis 
were different from those for the other tools 
PolySpace Verifier looks for run-time errors, e.g., 

- un-initialized variables/pointers 

- out-of-bound array accesses 

- overflow/underflow 

The original C++ code was translated into C code 
instead of Java 

The tool had to be run overnight in a batch mode 
because of its slow performances 


times Heseere!! Center- 


Model Checking 
Observations 


Asked never to “run” the code, only model check it 

* Keep the results dean from any testing influence 

Performed much better than testing, and, as well as runtime-analysis 

- Missed one concurrency error (nobody found this one) and one plan error 
Interesting observations 

- Partially abandoned the time abstraction within the first hour for one that is 
closer to real-time, but might miss errors 

• It was too hard for them to determine if errors were spurious not knowing the code 
well enough 

•- Didn’t use the Universal planner as much as we anticipated 

• Rather change the training plans we gave them, probably to be more in control 

- Lots of time spent with the heuristic options 

* The state space is very large and heuristics were required to look at different parts 

- Found a number of bugs In the first version, had a slow 2 nd version, and then 
found all the remaining bugs in the first part of the version 

• Took them some time to get their framework setup, but once done, they were flying 

- Found a nasty bug in floating-point arithmetic that slowed them down at the 
end 

* We anticipated a lot more tool errors than actually happened 18 


<0 

^JmsBssmBSsnlsr- 


Static Analysis 
Observations 


A priori static analysis seems easy to use: 

- You feed die program to the analyzer, and out comes a 
list of errors and warnings you can easily sort through 

The experiment shows that it is not that easy: 

- Participants didn’t understand how to deal with 
warnings - there are many more warnings than errors 

- It is difficult to understand how approximations in the 
analysis algorithms impact warnings, unless one has a 
good understanding of the algorithms 
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/ hmzs Besemft Heater- 


Static Analysis 
Observations cont. 


The domain of applicability of each operation 
flagged as an orange (warning) should be checked 
in every possible execution context 

- There are too many warnings to do this rigorously 

- Participants didn’t understand how, and where, to use 
assertions and stubs to eliminate oranges 

The participants chose to increase the number of 
execution paths that could he analyzed instead of 
analyzing the given program 

- They tried to make dead code reachable 


WA 


Runtime Analysis 


/ BmsHssesrvtt Center - 


* Java PathExpIorer 

- Required no setup. Instrumentation is automated. No 
specification or program manipulation is required. 

• DBRover 

- Rover code was pre-instrumented to emit events of 
the form (for actions V and time points Y): 

* start(a,t), success(a,t) and fail(a,t). 

- Users had to write a set of temporal formulae for each 
plan. This was time consuming. 


Java FathKxpiorer Jg 

Deadlock and Data race Detection ^8 

mesBese&sk Heater * — — — — 

6 Students quickly learned to use tool. Interrelation 
of results required some training. 

• Users found it very easy to apply tool They 
applied it instantly when they got a new version of 
the code, and then with regular intervals. 

* Tool found all seeded resource deadlocks and the 
seeded data race, and quickly. Tool is not designed 
to find communication deadlocks. Did therefore 
not find any. 

* No false positives or false negatives. 

• Extension of tool to handle some communication 
deadlocks is under way. 


yg JDUKover Hg 

Temporal Logic Monitoring 

Mbs H emrch Ssitter — — - — — — ■ — — ■■■ . ■ ■■ — 

• Students quickly learned to use DB Rover. Perhaps 
because temporal properties followed templates. 

• Users fqund it siightly inconvenient to write specs 
for new plans. DBRover was therefore only used 
sporadically. 

• Most plan errors (excluding deadlocks and data 
races) were found by examining printed 
information. Some were found due to violated 
temporal properties. 

• Automatic generation of specs from plans would 
have made DBRover a clear success (in the users 
own words). In particular, combined with a 
universal planner. This work is now being; done. 
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General Problem 



* Model checking and static analysis do best with 
mechanical (non-functional) properties 
' - Model checking: concurrency errors, such as deadlock 


and data races 

- Static analysis: runtime errors, such as, null pointer 
dereferences, array out of bounds, etc. 

• These are the “simple” bugs, but the real problems 
will come from the functional defects 

• Suggestion 

- Use model checking and static analysis to derive 
“good” test inputs and then use advance runtime 
monitoring during testing 


Autonomy Sisk Model 

VMesB&mrsh tenter - — 



* V&V risks and mitigations 

* Project Plan 

- V&V survey to find what the autonomy experts think 
are the risks and current mitigations 

• Model verification, large environments, etc. 

- Case studies on autonomy software 

• Real autonomy code seeded with typical bugs 

• State of the art V&V tools 

- Use results to populate risk models 

• AUTONOMO (based on COCOMO & COQUALMO) 
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Autonomy V&V 



• On-board autonomy 

- Remote Agent experiment a success 

- Mission managers are still skeptical 

• Planning & Scheduling on Earth used to schedule 
Mars Exploration Rovers daily activities 

• New projects 

- “A Model of Cost and Risk for Autonomy” 
t- “Verifying Autonomy Software” 
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